Chapter 1: Concepts and definitions

Goals

Understand conceptual and operational definitions

Know some strategies for clarifying conceptual definitions and writing conceptual definitions

Defining terms

Research Questions

  • Does religion play a role in civil wars?

  • What causes increasing income inequality in least developed countries?

  • Why has public opinion on same sex marriage liberalized so rapidly?

  • Is the U.S. becoming more polarized?

Not everyone is going to agree on what these terms mean!

Before we can get anywhere researching these questions, we need to define the thing we’re studying

Defining things: harder than it seems!

Reasonable definition of a chair

Chair

Terms

Conceptual Definition

A description of the concrete, measurable properties of a concept and the unit of analysis to which it applies

Unit of Analysis

The entity that that is being studied. For instance: individuals, governments, parties etc.

Operational Definition

A description of the instrument used to measure the concept

Crafting Conceptual Definitions: Unit of Analysis

  • What is the unit? What entity possesses the characteristic?

    • The U.S. and Canada are democracies (concept: democracy, unit: countries)

    • Lincoln, LBJ, and Trump are the three tallest U.S. presidents (concept: height, unit: presidents/people)

    • Labour is the UK’s main center-left political party (concept: party ideology/family, unit: political party)

One way to think of this: if you put your data into a spreadsheet, what does each “row” represent? The answer to that question is the unit of analysis.

Crafting Conceptual Definitions: Key features

  • What are the essential features of the concept?

    • If there are cases or definitions everyone agrees on, what characteristics do they share?

    • Are certain qualities necessary or sufficient for a case to belong to a category?

    • What characteristics are most distinctive to those cases?

    • What is most helpful for clarifying edge-cases?

  • Are there multiple dimensions or just one? If two characteristics always occur together, you might only need to account for one of them!

Crafting Conceptual Definitions: transparency

We want to avoid being Potter Stewart:

The goal is cumulative knowledge, and a private definition can’t provide that.

Some steps to take:

  1. Identify the unit of analysis

  2. Make a list of important properties, clear examples and non-examples, and/or generally accepted definitions

  3. Remove items that aren’t measurable

  4. Reduce dimensions where possible (if a characteristic is shared by positive cases and negative cases, then it isn’t useful)

  5. Refine as needed

Conceptualizing Democracy

Some proposed features:

  1. Regular elections with meaningful alternatives

  2. Peaceful transfer of power

  3. Free expression

  4. A competitive media environment

  5. Autonomous political groups (political associations, unions, interest groups etc.) that can pressure the government

  6. Rule of law

  7. Checks and balances

  8. Property rights

Minimalists might argue that 1 and 2 alone are necessary and sufficient. Others may argue that even these 8 are oversimplifying.

Conceptualizing Democracy

  • Unit of analysis: governments (usually national governments)
  • Cases:
    • Definitely democracies: U.S., U.K., Uruguay, Taiwan, Japan, Botswana
    • Definitely autocracies: North Korea, Saudi Arabia, Russia, Venezuala
    • Edge cases: Hungary, Turkey, Tunisia
  • Features:
    • Voting and elections
    • Multiple parties
    • Meaningful civil liberties
    • Stability and monopoly on violence

This may need refinement: what about sham elections?

This may not be distinctive: there are stable autocracies on the list

Minimalists might argue that we only need the elections and parties

Operational Definitions

Operationalization

Even where we agree on a definition, we will need to measure a concept and there is often slippage here

Considerations: Parsimony

In many cases, we trade some truth for simplicity

This is a very accurate rendering of the DC metro area

But this is probably more useful for getting around.

Considerations: Parsimony

Similarly: we might consider “democracy” to be a highly complex multidimensional concept

mindmap
  root((Democracy))
    Liberal Rights
        Free Speech
        Free Association
        Rule of law
    Deliberation
        Respectful Dialogue
        Adversarial Press
        Broad Participation
    Egalitarianism
        Equal Rights
        Equitable access to resources
        Diverse representation
    Majoritarianism
        Fair elections
        Peaceful transfer of power
        Meaningful alternatives

Considerations: Parsimony

…but we might still prefer an operational definition that relies on a subset of concepts that are easy to measure.

mindmap
  root((Democracy))
    Majoritarianism
        Fair elections
        Peaceful transfer of power
        Meaningful alternatives

Operationalizing Democracy

Dictatorship and Democracy (Alvarez, Cheibub, Limongi, & Przeworski 1996)

Dichotomous measure. Democracies have the following:

  • Have popular elections to fill seats

  • Have more than one party

  • The incumbent sometimes loses

Varieties of Democracy (V-DEM) Project (link)

Democracy along multiple dimensions. Scores are determined by:

  • Polling a large group of area experts on a range of democratic dimensions (electoral, deliberative, liberal etc.)

  • Using an algorithm to aggregate those responses into scores along each dimension.

Both of these approaches have acknowledged trade-offs. The minimalist view has the advantage of parsimony, but the more nuanced measures may get us closer to the ideal conceptual definition.

But neither is free from error.

Also note, these are influenced by different conceptual definitions, which are in turn potentially influenced by some normative considerations.

Considerations: Reliability

Reliability refers to how consistently the same measurement instrument produces the same result

  • Test-Retest method: asking the same questions to a new random sample should produce roughly the same result

  • Split half: for a measurement using a multi-item scale, we can split results and use half the questions to predict the remaining half.

(One purported advantage of the minimalist definition of democracy is that its relatively stable because it allows less room for subjectivity.)

Considerations: Validity

  • Validity refers to whether an operational definition matches the concept we’re interested in.

  • A challenge to validity might come from cases seem miscategorized in our definition. For example: the minimalist definition of democracy suggests that Japan was authoritarian for much of the latter half of the 20th century.

Measurement Error

  • Random Error

    • Interference

    • Data entry errors

    • Random non-response

  • Systematic Error

    • Hawthorne effect

    • Social desirability bias

    • Systematic non-response (surveys) or missing data (everything)

Measurement Error

  • Measurement error is a source of noise, systematic measurement error is a source of bias

  • We’ll generally find that random noise is much easier to deal with than bias because we can simply collect more data. Bias, on the other hand, presents profound challenges to research because we can only correct it if we can measure it

Measuring Party ID

Consider measuring party identification:

Instrument Problem
Registration Excludes new/non-voters, people in states without partisan registration, and people who register one way and vote another
Policy views Complicated to measure, and a surprising number of people don’t have consistent policy views!
Voting Behavior Excludes new/non-voters, may not be consistent even in a single election, people may exaggerate or forget voting behavior.
Self Description* Subjective, far more self-described “independents” than people who consistently vote that way

Party ID

Party ID

Party ID

Party ID

  • The ANES method for measuring party ID is hardly the only option we have, but it has some nice features:

    • Its relatively reliable and survey respondents understand it

    • Its close to the conceptual definition of party ID as a sort of self-affiliation

    • Its transparent and widely used

    • It limits the influence of social desirability bias by giving respondents lots of options

    • Its linked to observed behavior: vote choice

Key points

  • Asking social science questions requires us to define complex ideas and measure them. And we can lose a lot in that process.

  • There are some wrong answers, but there is generally no single correct answer. We generally have to make trade-offs between things we might value like:

    • Parsimony

    • Reliability

    • Validity

    • Probability of random or systematic errors

  • We can’t really achieve perfection, but where there is disagreement, we want to be transparent about methods and limitations.

Reminders

  • Install R and R-Studio for Friday discussion

  • Start thinking about questions for pilot design